The present invention relates generally to the field of optical microscopy and, in particular, to a new and useful adaptive scanning optical microscope which addresses and improves upon, the usual trade-off between resolution and field of view that is common for known optical microscopes.
The inventors have disclosed an earlier approach to solving this trade-off problem in U.S. patent application Ser. No. 10/525,422 filed Feb. 25, 2005. This application claims priority on U.S. Provisional Patent Application No. 60/411,038 and International Application No. PCT/US2003/029332, published as WO 2004/025331, all of which are also incorporated here by reference.
For a wide range of applications (e.g. micro-assembly, biological observation, observation and manipulation for biotechnology, medical diagnostics, manufacturing, inspection, etc.), the optical microscope remains one of the most important tools for observing below the threshold of the naked human eye. However, in its conventional form, it suffers from the trade-off mentioned above, between resolution and field of view. The present invention is a new optical microscope design that combines a scanner lens, a steering mirror, an adaptive optics element, adaptive optics (AO) conditioning optics, and imaging optics to enlarge the field of view while preserving resolving power in the acquired images. This instrument has the ability to operate at high image acquisition rates for increased throughput or to facilitate certain spatial-temporal observations.
Along with the recent growth of biotechnology and micro-electro-mechanical systems (MEMS), as well as an industrial trend toward miniaturization, there is a growing need to observe, interact with, and inspect at a scale below the threshold of the naked human eye. Fulfilling this need, the optical microscope has seen a resurgence of interest and will continue to be a critical tool as these fields advance. However, the essential optical design and operating principle has not changed significantly in the last century, and the optical microscope still suffers from a well known inherent tradeoff between the field of view and resolving power of the imaging system.
The present invention, as will be explained more fully later in this disclosure, achieves an expanded field of view at high resolution by integrating active optical elements, motion control, and image processing techniques with traditional static optical elements in a tightly integrated fashion.
The motivation for expanding the field of view initially came from the inventors' experiences in micro-assembly and precision manufacturing. Vision guided micro-assembly often requires the near-simultaneous monitoring of widely separated part features at micron to sub-micron level resolution (e.g. monitoring multiple critical edges of a micro-mirror and optical sensor being assembled onto a substrate). Because a single microscope can not offer an adequately large field of view at the required resolution, multiple microscopes and/or a moving stage provide a readily available off-the-shelf solution. However, the limitation in movements per second and agitation of the specimen due to the moving stage, and considerable effort required to reposition and calibrate multiple microscopes for each new assembly task, suggested to the inventors, a need for a new optical microscope design to address these issues.
For the same reasons, such a microscope would also be desirable for biological and medical imaging as well as industrial manufacturing and inspection as is performed using machine vision. The inventors first design which was disclosed in their earlier U.S., PCT and provisional patent applications identified above, and in the article B. Potsaid, Y. Bellouard, and J. T. Wen, “Scanning optical mosaic scope for micro-manipulation,” in Int. Work-shop on Micro-Factories (IWMF02), R. Hollis and B. J. Nelson, eds., pp. 85-88 (2002), was called a Scanning Optical Mosaic Scope (SOMS), and was constructed to demonstrate the advantages of combining a high speed post-objective scanning system with real-time mosaic constructing techniques for use in micro-assembly and biological imaging. The optical layout for the previously disclosed SOMS was originally inspired by a machine created for laser annealing shape memory alloy. See M. Hafez, Y. Bellouard, T. Sidler, R. Clavel, and R.-P. Salathe, “Local annealing of shape memory alloys using laser scanning and computervision,” in Laser Precision Microfabrication, I. Miyamoto, K. Sugioka, and T. Sigmon, eds., Proc. SPIE 4088, pp. 160-163 (2000). This approach shares the concept of a post-objective 2-D scanning mirror. This configuration is also used in several commercial products, but in its basic form, has a limited field of view because of off-axis aberrations in the scanner lens. The present invention addresses this issue to offer a larger field of view.
The design of wide field and high resolution microscopic imaging systems are driven by consideration of (1) an image sampling issue and (2) an image quality issue. First, consider an imaging system with optics that are nearly perfect (i.e. the optical aberrations are much below the diffraction limit). Such a system will image two point sources separated by a distance, d, as two overlapping Airy patterns in an image field. As the distance between the two points decreases, a critical distance will be reached, r, where the two points can no longer be individually distinguished. According to the Rayleigh criteria, this critical distance, called the resolution, occurs when the center of one Airy disk falls on the first minimum of the other and is related to the numerical aperture, NA, of the system and the wavelength of light, λ. The NA of the system is a function of the index of refraction of the transmitting medium, n, and the half angle of the cone of light collected from the object.
A digital camera must sample with two pixels per Airy core radius to avoid aliasing according to the Nyquist sampling criteria. This observation provides a maximum theoretical object field width, Wo, for a sensor array pixel count per edge, k, and resolution, r.
While microscopic imaging systems are often designed with resolutions in the ¼ μm to several μm range, the lower practical limit on CCD camera pixel size is approximately 6 μm due to noise effects. Therefore, the optics must enlarge the Airy pattern to achieve proper sampling, with the required minimum magnification factor, M, for a given sensor pixel size, s. At this critical magnification, the corresponding image size, Wi, is: Wi=ks. Imaging optics to achieve this can be thought of as a generic black box. The optical design task is to specify the design of the imaging system, i.e., to fill in the details of the black box with specific lens or mirror geometries, glass types, and spacing.
An intuitive approach to designing a large field and high resolution imaging system might be to take an existing microscope layout, and simply increase the pixel count of the camera while redesigning the optics to achieve a larger field of view. This approach may indeed be possible, but it is not generally practical as the requirements for field size, flat field, and numerical aperture soon approach those of lithography lenses. The 1998 Nikon lithography lens (see U.S. Pat. No. 5,805,344 for example) has a 0.65 NA with field sizes of 93.6 mm and 23.4 mm for the mask and wafer image respectively.
Lithography lenses require near perfect manufacturing and extremely tight assembly tolerances (often requiring an interferometric assembly process), and can cost in the millions of dollars. Also, negatively powered elements are required and are located at narrow beam regions in both the microscope and lithography lenses and positively powered elements where the beam is wide. This design technique is used to achieve a flat imaging field (small Petzval sum) and results in an increase in the lens count and optical complexity. An additional consideration is the size of the image sensor, given that large commercially available CCD cameras only have approximately 9216×9216 pixels (e.g. Fairchild Imaging CCD595). Smaller CCD arrays can be assembled into a mosaic to achieve larger pixel count with the advantage of being able to read data off the imaging chips in parallel (data rates forgetting the image data off the chip can be the limiting factor determining maximum refresh rates), but at a cost of additional precision assembly requirements. Even with modern technology and manufacturing capabilities, a large field and high resolution imaging system based on a purely static optical design will only see limited application because of the exceedingly high cost, large size, tight assembly tolerances, and optical complexity.
Some of the alternative modern approaches to address the field size and resolution tradeoff are summarized in Table 1 which includes the performance of the present invention for comparison. The first five methods (multiple parfocal objectives through multiple microscopes) are well established and quite common. In this table, the “basic post-objective scanning” method refers to the commercially available units, which are limited to very low numerical aperture and suffer from considerable off-axis aberration because of the system layout. Of particular interest is the array microscope sold by Dmetrix. Dmetrix is covered by several patents, for example: U.S. Pat. No. 6,958,464, for an Equalization for a multi-axis imaging system; U.S. Pat. No. 6,950,241 for a Miniature microscope objective for an array microscope; U.S. Pat. No. 6,905,300 for a Slide feeder with air bearing conveyor; and U.S. Pat. No. 6,842,290 for a Multi-axis imaging system having individually-adjustable elements. This system uses an array of 80 miniature microscopes (each of 3 element aspheric design) working in parallel to rapidly acquire the image. By slowly advancing the microscope array along the length of a microscope slide, a large composite image can be constructed. Given the parallel imaging paths, this is the fastest area scanning technology producing medical diagnostic grade images of static objects that the inventors are aware of at this time (scanning, compressing, and storing an area of 225 mm2 at 0.47 microns per pixel in 58 seconds). A related technology is the line scanning system, which sweeps a specimen (often projected through a microscope objective) past a linear array of sensor pixels. A major disadvantage of line scanning technology is that images are obtained line by line (n×1 pixels) as opposed to area by area (n×n pixels), as is the case with a more typical area based image sensor. A consequence is that line scan systems generally require extremely short exposure times and/or bright illumination to obtain high throughput, which is often not possible in biological applications where photo-damage, bleaching, and fluorescence must be considered.
With parallel image acquisition and a relatively slow re-positioning speed, the DMetrix excels at static and high fill factor applications. Fill factor is the percentage of the total observable area that is of interest and absolutely must be imaged or sensed for the application at hand. Because the ASOM of the present invention acquires images serially in time with extremely fast re-positioning speeds, the ASOM will excel in dynamic and/or low fill factor applications. Low fill factor applications include biological imaging of rare events over a large cell population, tracking multiple moving organisms, medical diagnostics of tissue sampled by needle extraction which is haphazardly placed on a microscope slide, etc. Most manufacturing applications require a low fill factor as only certain critical regions need to be observed or inspected with dynamic tracking of objects or features often required during assembly.
More generally, the ASOM of the present invention is particularly suitable for challenging spatial-temporal observation tasks requiring both a wide field of view and high resolution. Consideration of these issues motivated and contributed to the design of the ASOM.
TABLE 1Qualitative comparison of Present Invention (ASOM)to other technologies.PreservesEasyScanningresolvingmanufacturingrateEasilyNo specimenpower whileintegration over(movementsreconfig-uredIlluminationagitation duringexpandingconveyoror imagesfor differentbrightnessscanningfield of viewtransportper second)viewing tasksrequirementsMultipleXLOWNORMALParfocalObjectivesZoom LensXXMEDNORMALDesignMoving StageXMEDXNORMALMovingXXXLOWXNORMALMicroscopeMultipleXXXHIGHNORMALMicroscopesBasic Post-XXHIGHXNORMALObjectiveScanningDmetrixXXHIGHNORMALLine ScanningXXHIGHVERY HIGHASOMXXXHIGHXNORMALpresentinvention
Adaptive optics technology with deformable mirrors have been used to allow for high resolution imaging inside the human eye (see H. Hoffer, L. Chen, G. Y. Yoon, B. Singer, Y. Yamauchi and D. R. Williams, “Improvement in retinal image quality with dynamic correction of the eye's aberrations,” Opt. Express 8, 631-543 (2001), http://www.opticesxpress.org/abstract.cfm?URI=OPEX-8-11-631), which is particularly challenging because of the time varying aberrations of the eye's lens. Similarly, deformable mirrors have also been used to correct for off-axis aberrations and sample induced wavefront disturbances in confocal microscopy. Expanding the field of view in imaging systems has also previously been shown with a liquid crystal spatial light modulator to create a foveated imaging system (see D. Wick, T. Martinez, S. Restaino, and B. Stone, “Foveated imaging demonstration,” Opt. Express 10, 60-65 (2002), http://www.opticsexpress.org/abstract.cfm?URI=OPEX-10-1-60).
A confocal microscope of know design, uses a pinhole screen located in a plane that is conjugate to the object plane. This pinhole rejects light that is not at the same depth as the focal plane. The pinhole also rejects light that is not at the center of the field. Thus, the confocal microscope samples the object point by point. The image is built up point by point and there is a means for scanning the location of the imaging point on the specimen. A basic introduction to the confocal microscope can be found at: http://www.physics.emory.edu/˜weeks/confocal.
The ASOM of the present invention acquires images using finite imagery (i.e. an entire 2 dimensional image is exposed all at once rather than building up an image point by point). The requirement to perform finite imagery imposes certain requirements on the optical system that are not necessary for point sampling techniques (confocal microscopy). Some of the advantages of the ASOM's finite imagery based approach are that multiple regions of the object are imaged in parallel, resulting in faster acquisition times. This is particularly important for low light conditions or when the object is in motion. There are also advantages with respect to the illumination requirements. However, a finite imagery based system does not offered ability to vertically “section” the sample as a confocal system can.
U.S. Pat. No. 6,771,417 discloses a non-confocal arrangement that includes adaptive optics. See U.S. Pat. No. 6,555,826 for a confocal arrangement including adaptive optics and U.S. Pat. No. 6,381,074 for an adaptive optics element in a scanning confocal microscope to assist in aberration control and precise focusing. U.S. Pat. No. 6,483,641 discloses a spatial light modulator used in a microscope.